\section{Discussion}
\label{sec:discussion}

ESJ provides a general join operator for big streaming data, having the following properties.

%%% Claim 1 %%%
\begin{claim}
ESJ is independent of the applications.
\end{claim}

By design, ESJ is application-agnostic since it applies comparison-based join processing, i.e., evaluating the join predicates by comparing the join key values. Thus it is unaware and independent of the content of the input streams.

%%% Claim 2 %%%
\begin{claim}
ESJ supports theta-join and multi-attribute join.
\end{claim}

ESJ supports both equality and inequality join predicates, using hash map and BST to accelerate the processing, respectively. Compound multi-attribute predicates are also supported. One predicate is chosen to be the \textit{main predicate} which is processed through BST or hasp index. Remaining predicates are treated as \textit{secondary predicates} which are processed by piggybacking the processing of main predicate. Consequently, multi-attribute join predicates of one query can be processed in single join task with the same asymptotic time as in the processing of unitary predicate. Furthermore, if the domain distribution of input streams is given as statistics, ESJ could leverage it to select the main predicate such that the join keys have least skewed distribution, which benefits the efficient join processing. Besides, ESJ can further support streaming self-join by duplicating the onefold streaming input into two so that its processing is normalized to that of the binary streaming theta-join.

%\begin{figure}[t]
%\centering
%\epsfig{file=pic/self_join.eps, width=0.8\linewidth}
%\caption{Streaming Self-Join Processing by ESJ.}
%\label{fig:self_join}
%\end{figure}

%Besides, ESJ can further support streaming self-join by duplicating the onefold streaming input into two so that its processing is normalized to that of the binary streaming theta-join (Figure~\ref{fig:self_join}~(a)). Moreover, if the self-join windows do not overlap, duplicating the stream could be unnecessary since ESJ could reuse the expired stream as the opposite streaming input (Figure~\ref{fig:self_join}~(b)).

%%% Claim 3 %%%
\begin{claim}
ESJ supports concurrent join tasks on an input stream pair.
\end{claim}

ESJ supports running multiple join tasks simultaneously upon the same stream pair. Each join task is processed by a set of cascading worker instance. All worker instances belonging to the same join worker share the stream pair so that no duplication is needed for this stream pair. To handle the scaling number of concurrent join tasks, worker instances are allocated and deallocated in the cluster, and distributed to achieve load balancing. In a source-sharing multi-task join application, more tasks will lead to more saving of communication cost due to no stream duplication for every join task.

%%% Claim 4 %%%
\begin{claim}
ESJ is elastic to the workload of join tasks.
\end{claim}

% ESJ benefits from the elasticity management for effectively handling big streaming data.

The increase of workload may exceed the processing capability. Large workload results from the high incoming rate of input streams, the big size of join window, or their combination. To handle the high incoming rate and the large join window, we could either add more workers or apply load shedding. This is a trade-off between the accuracy and the resource availability.

%%% Claim 5 %%%
\begin{claim}
ESJ provides a strict real-time guarantee for join processing.
\end{claim}

Real-time processing requires that the submitted task must be done in the deterministic time span. As streaming join is constrained by a finite sliding window, the output for each input is expected to be within the same time span. ESJ joins each incoming tuple with all valid matching within the join window. Each tuple is assigned a lifespan when it enters the ESJ processing topology. The lifespan equals to the join window size. Consequently, active tuples will be joined with the matching tuples within its lifespan, while the expired tuples will be evicted from the processing topology. As long as the tuple is expired, all join results related to it have already been outputted. Therefore, the join processing time with respect to a tuple is guaranteed to be real-time.

Moreover, the control of join window should not be the bottleneck of the overall performance. Centralized join window control suffers from high communication cost, and it may become the straggler when dealing with big window size. ESJ applies the dataflow-oriented processing so that the maintenance of join windows is amortized to every streaming data transfer without any centralized control. The join window size could be adjusted on the fly without resubmitting the query. This is done by changing the lifespan assignment for new coming tuples since the join window size is reflected as the lifespan.

% section discussion (end)